from sklearn_benchmarks.reporting.hp_match import HpMatchReporting
import pandas as pd
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.018	0.154	0.000	0.002	-1	1	0.660	19.916	0.063	0.660	0.101	0.101	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1	100	0.025	0.003	0.000	0.025	-1	1	0.000	0.368	0.008	0.000	0.068	0.068	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.014	0.065	0.000	0.003	-1	5	0.752	19.859	0.077	0.752	0.152	0.152	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.027	0.004	0.000	0.027	-1	5	1.000	0.364	0.007	1.000	0.074	0.074	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.268	0.015	0.000	0.002	1	100	0.877	19.761	0.040	0.877	0.115	0.115	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.000	0.000	0.021	1	100	0.000	0.361	0.009	0.000	0.058	0.059	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1000	100	3.038	0.053	0.000	0.003	-1	100	0.877	20.322	0.039	0.877	0.150	0.150	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.026	0.002	0.000	0.026	-1	100	0.000	0.361	0.008	0.000	0.072	0.072	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.241	0.008	0.000	0.002	1	5	0.752	19.753	0.057	0.752	0.113	0.113	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	1	5	1.000	0.356	0.008	1.000	0.059	0.059	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.204	0.018	0.001	0.001	1	1	0.660	19.858	0.065	0.660	0.061	0.061	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	1	0.000	0.357	0.008	0.000	0.055	0.055	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.753	0.038	0.000	0.002	-1	1	0.962	4.227	0.033	0.962	0.415	0.415	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1	2	0.005	0.001	0.000	0.005	-1	1	1.000	0.284	0.005	1.000	0.016	0.016	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1000	2	3.011	0.069	0.000	0.003	-1	5	0.975	4.188	0.038	0.975	0.719	0.719	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	2	0.008	0.002	0.000	0.008	-1	5	1.000	0.289	0.009	1.000	0.028	0.028	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.215	0.013	0.000	0.002	1	100	0.981	4.220	0.023	0.981	0.525	0.525	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	100	1.000	0.277	0.005	1.000	0.010	0.010	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.923	0.039	0.000	0.003	-1	100	0.981	4.214	0.022	0.981	0.694	0.694	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	2	0.007	0.005	0.000	0.007	-1	100	1.000	0.283	0.006	1.000	0.025	0.025	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.199	0.013	0.000	0.002	1	5	0.975	4.119	0.012	0.975	0.534	0.534	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1	2	0.003	0.000	0.000	0.003	1	5	1.000	0.281	0.005	1.000	0.010	0.010	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.076	0.007	0.000	0.001	1	1	0.962	4.214	0.010	0.962	0.255	0.255	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.002	0.000	0.000	0.002	1	1	1.000	0.285	0.013	1.000	0.007	0.007	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.869	1.394	0.000	0.004	-1	100	0.957	132.189	0.000	0.957	0.029	0.029	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.008	0.001	0.000	0.008	-1	100	1.000	3.081	0.253	1.000	0.003	0.003	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	6.709	1.659	0.000	0.007	1	100	0.957	130.995	0.000	0.957	0.051	0.051	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	1	100	1.000	3.087	0.280	1.000	0.002	0.002	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.133	0.560	0.000	0.001	1	1	0.942	131.355	0.000	0.942	0.009	0.009	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	1	1.000	3.088	0.249	1.000	0.000	0.000	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.119	0.388	0.000	0.001	-1	5	0.966	130.419	0.000	0.966	0.009	0.009	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	-1	5	1.000	3.041	0.183	1.000	0.001	0.001	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.591	0.140	0.000	0.001	-1	1	0.942	132.429	0.000	0.942	0.004	0.004	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.004	0.001	0.000	0.004	-1	1	1.000	3.132	0.335	1.000	0.001	0.001	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.976	0.434	0.000	0.002	1	5	0.966	132.800	0.000	0.966	0.015	0.015	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.002	0.000	0.000	0.002	1	5	1.000	3.118	0.249	1.000	0.001	0.001	See	See
12	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.054	0.022	0.000	0.000	-1	100	0.915	0.076	0.014	0.915	0.710	0.721	See	See
13	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.004	0.002	0.000	0.004	-1	100	1.000	0.006	0.000	1.000	0.676	0.677	See	See
14	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.001	0.000	0.000	1	100	0.915	0.073	0.002	0.915	0.558	0.558	See	See
15	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	100	1.000	0.006	0.000	1.000	0.111	0.111	See	See
16	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.021	0.000	0.001	0.000	1	1	0.890	0.045	0.001	0.890	0.477	0.477	See	See
17	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	1	1.000	0.006	0.000	1.000	0.106	0.106	See	See
18	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.028	0.001	0.001	0.000	-1	5	0.912	0.045	0.001	0.912	0.630	0.630	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	5	1.000	0.006	0.000	1.000	0.438	0.439	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.025	0.001	0.001	0.000	-1	1	0.890	0.044	0.001	0.890	0.574	0.574	See	See
21	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.003	0.000	0.000	0.003	-1	1	1.000	0.006	0.000	1.000	0.415	0.415	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.024	0.001	0.001	0.000	1	5	0.912	0.046	0.002	0.912	0.515	0.515	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.001	0.000	0.000	0.001	1	5	1.000	0.006	0.000	1.000	0.113	0.113	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

Raw results

predict

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.129	0.001	300	0.006	0.00	0.825	0.553	0.023	0.825	0.234	0.234	See	See
1	HistGradientBoostingClassifier_best	predict	100000	1	100	0.020	0.001	300	0.000	0.02	1.000	0.451	0.006	1.000	0.043	0.043	See	See